Raw camera peripheral for handheld mobile unit

ABSTRACT

A handheld imaging device is described to provide high quality RAW image data to a smartphone, or similar handheld imaging device. The imaging device can have a larger dimension, particularly with regard to the thickness of the device, thus overcoming some of the physical limitations to providing high quality optics and image sensing within the thin form factor of a modern smartphone. As the device is only connected when capturing images the smartphone form factor is not altered. The device does not replicate the high-end image processing functionality of a smartphone, but instead transfers RAW images to the smartphone for high-end image processing &amp; enhancement on the smartphone CPU/GPU. Thus it can be manufactured at lower cost than a dedicated camera with equivalent capabilities by taking advantage of the sophisticated image processing capabilities of today&#39;s smartphones.

CROSS-REFERENCE

Priority is claimed from Irish Pat. App. No. S2014/0135, filed Jun. 3,2014, and issued as Irish Pat. No. S86536 on Apr. 14, 2015; Irish Pat.App. No. S2014/0136, filed Jun. 3, 2014, and issued as Irish Pat. No.S86520 on Mar. 3, 2015; Irish Pat. App. No. S2014/0134, filed Jun. 3,2014, and issued as Irish Pat. No. S86537 on Apr. 14, 2015; and IrishPat. App. No. S2014/0133, filed Jun. 3, 2014, and issued as Irish Pat.No. S86519 on Mar. 3, 2015; all of which are hereby incorporated byreference.

FIELD

The invention is in the field of consumer imaging, more specifically animaging add-on device for a smartphone.

BACKGROUND

Modern smartphone devices have settled on a compact and thin format.While this is convenient for the user, both in terms of carrying thedevice in a pocket and holding it in the hand while in use as a mobilephone it creates difficulties in resolving a high-quality optical image.Thus while the camera modules in today's smartphones have continued toimproved in term of pixel resolutions, speed of image acquisition and awide range of digital manipulation of the underlying image the opticalquality is constrained by the physical limitations of the device formfactor and the corresponding size of image sensor that can beaccommodated within the device.

Some attempts have been made to improve the image quality by providingadditional lens elements that clip onto the device over the existingcamera lens to increase the zoom, enhance the field-of-view, or toprovide improved macro capabilities. However these optical add-on lensesare constrained to the design parameters of the original lens system andthe size of the internal sensor element of the smartphone (Dainty 2012).

Some manufacturers have created independent camera modules thatcommunicate with a smartphone via wireless communications such as Wifi(IEEE 802.11). These modules can be attached to the case of a smartphoneor may be operated completely independently of the smartphone. Theyexport a control user inferface (UI) for the camera and images arecaptured in the camera module and compressed to JPEG format (or MPEGformat for video) prior to being transferred wirelessly to thecontrolling smartphone unit.

While these devices can obtain high-quality compressed images and videoand may be controlled and operated from the smartphone, they do notsupport the acquisition of high quality RAW (Bayer) images. Furthermorethey require a high-end full imaging system, including dedicated imagesignal processor (ISP) and main system CPU/GPU and JPEG (images) or MPEG(video) compression engine to be provided within the independent cameramodule.

With current state of prior art we see that improved imaging can beachieved on handheld imaging devices such as smartphones either byadding (i) an enhanced clip-on lens, or (ii) by connecting a dedicatedand fully-independent camera module to the smartphone over a wireless(or wired) link. The first solution is limited by the original opticaldesign and image sensor of the smartphone. The second approach overcomesthese limitations but requires a full imaging pipeline and compressionengine in order to process and compress images that are suitable fortransfer over a wireless link.

Thus there is a need for an add-on peripheral that improves the opticalacquisition, and can accommodate a larger sensor size (APC orfull-frame) but that can also take advantage of the inbuilt imageprocessing capabilities and system-level ‘apps’ of the smartphone. Thereis a further need for advanced user interfaces enabling accurate andsimplified control of complex digital camera acquisition parameters byusers who are not familiar with the operation of advanced DSLR andmirrorless cameras.

SUMMARY

For a better understanding it is useful to consult US 2013/0004071 Imagesignal processor architecture optimized for low-power, processingflexibility, and user experience to Chang et al. This describes an imagesignal processor architecture that may be optimized for low-powerconsumption, processing flexibility, and/or user experience. In oneembodiment, an image signal processor may be partitioned into aplurality of partitions, each partition being capable of entering alower power consumption state. Techniques for the partitioning of an ISPas described in this patent application may be used advantageously inthe present invention.

An example ISP is shown in FIG. 2, taken from this patent application.It can be seen that the ISP as shown in FIG. 1 is significantly morecomplicated. In a practical embodiment the first partition of the ISPshown in FIG. 2, that is 202, would be equivalent to the Bayer ISP ofFIG. 1. The other functional elements of the ISP, that is 204 and 206,208 would be implemented on the host smartphone.

FIG. 3 shows the MIPI interface between a sensor and ISP, taken fromwww.mipi.org and based on CSI-2 variant. The latest practicalembodiments are known as M-Phy™ (physical layer) and UniPro™ (protocolstack). US-2013/0191568 Operating m-phy based communications overuniversal serial bus (usb) interface, and related cables, connectors,systems and methods to Hershko et al. describes how M-Phy (MIPI)interfaces can be controlled and data transfers achieved via a USB-3interface.

Thus a physical example of the present invention is presented in FIGS.4A and 4B which illustrates the camera module with external USBinterface. A smartphone with external USB connector can be attached ontop of said module and the module is then operated and controlled viathe USB interface. The internal details of the camera module areillustrated in FIG. 1 and the internal architecture of an exemplarysmartphone configured to connect to the device is shown in FIG. 5.

Various detail of this invention and in particular alternativeembodiments providing advanced user interfaces will be documented in thefollowing sections.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the RAW camera peripheral Module with the mainsub-components required for various embodiments.

FIG. 2 illustrates the various functions of an example Image SignalProcessor pipeline showing partitioning of functionality to allow moreenergy efficient operation (from US 2013/0004071). The present inventionemploys similar partitioning to allow separation of the RAW (Bayer)processing from higher level image processing functions.

FIG. 3: Shows a generic MIPI interface. MIPI is used by moststate-of-art handheld imaging devices to interface between camera-sensorand ISP; the CCI interface is a bidirectional command & controlinterface based on I2C signaling protocols, while the CSI interface is aunidirectional interface for offloading bulk image data from the sensor.

FIG. 4A: Camera module with USB connector and full-size lens module.

FIG. 4B: Camera module with USB connector and full-size lens module.

FIG. 5: Smartphone configuration to interface with camera module.

FIG. 6: Exemplary smartphone interface used to control camera settings(ISO, EV number, aperture and ‘equivalent’ exposure time) available insmartphone camera app.

FIG. 7A: RAW module with interchangeable hand-grip; and configured toreceive smartphone in a landscape orientation.

FIG. 7B: RAW module with interchangeable hand-grip; and configured toreceive smartphone in a landscape orientation.

FIG. 8: Exemplary Smartphone User Interface for RAW Camera Peripheral.

FIG. 9: Main acquisition control button with multi-functional rotarysliders providing convenient access to main acquisition functions.

FIG. 10: Extended acquisition control button overlaid on main live-viewimage, enabling convenient access to multiple acquisition settings andproviding guide to correct acquisition conditions.

FIG. 11: Detailed view of extended acquisition control button showingaccess to multiple acquisition settings.

DETAILED DESCRIPTION OF SAMPLE EMBODIMENTS

In a preferred embodiment the camera module is configured with afull-size DSLR or APS-C image sensor and incorporates a lens mountcompatible with a DSLR manufacturer. This allows the module to use anylens design from that particular camera maker. In addition the modulewill provide various electrical and mechanical interfaces as required tooperate the lens type(s) of that manufacturer. For example, U.S. Pat.No. 8,452,170 Digital single lens reflex camera to Hwang et al describesdetails of one such exemplary interface. This document, U.S. Pat. No.8,452,170, is incorporated by reference.

The camera module will also include exposure, auto-focus, shutter(capture) and zoom control subsystems compatible with the lens types ofthe relevant manufacturer.

The Raw Camera Module Physical Embodiments

The main embodiment is shown in FIG. 4A with a rear-view in FIG. 4B. Analternative embodiment is shown in FIG. 7.

The RAW camera module comprises a main body [401] which is of greaterdepth than the smartphone and is provided with clamps, bindings or otherfixing mechanisms (not shown) to facilitate connecting the smartphone tothe module in a robust manner. The smartphone will be connectedelectrically to the module via a USB-3 or similar high-speed data port[405] and in the embodiment shown in FIG. 4A will be mounted in aportrait orientation.

In certain embodiments the RAW module also features a lens mount [407]that facilitates interchangeable lenses [409]. Electrical connectors arealso provided in the lens mount to enable on-lens switches and controlsto provide settings information to the lens control subsystem [109] ofthe RAW module (shown in FIG. 1).

In certain embodiments the RAW module includes a hand grip (FIG. 7A)that improves handing of the device. This handgrip may be detachable sothat it can be switched from left-hand to right-hand side and thusfacilitate use of the module by left-handed photographers. In otherembodiments the module may be configured to allow the smartphone deviceto be connected in a landscape orientation allowing a widescreenuser-interface to be provided.

Internal Organization of the Module

This is shown in FIG. 1. A lens [110] is typically provided as aseparate component that can be removed and replaced with another lenswith different properties. As with digital single lens reflex (DSLR)cameras this allows a variety of wide-angle, macro and telephoto lensesto be used to acquire images in different photographic situations. Assuch lenses are relatively large it does not make sense to adapt asmartphone to accommodate removable lens assemblies, but the RAW moduleenables the use of such lens assemblies when the smartphone isconnected.

In addition to the lens assembly the RAW module also incorporates a lenscontrol [109] (and optionally a lens driver/motor, not shown) and focusand exposure control subsystems [114]. The purpose of the lens controlis to drive the movable lens elements as required by the focussubsystem.

In some embodiments a phase focus subsystem may be used and this mayinclude an additional sensing element. Alternatively a simple contrastbased focusing system may be employed and the focus subsystem will useeither RAW data direct from the image sensor, or may use the ISP topre-process certain RAW image data. In any case the contrast based focussubsystem uses selected portions of the full image scene and measuresthe contrast levels within these regions to determine the effects ofchanging the lens position. A range of other focus sensing approachesare described in the literature and may be adopted here.

The lens assembly may also include an aperture ring that allows changingthe size of aperture that admits light into the camera and onto theimage sensor [112]. It is also possible to adjust the time intervalduring which image data is accumulated by the pixels of the imagingsensor. Typically this will be a CMOS sensor and controlling thecombination of lens aperture and the accumulation interval for imagedata by pixels of the sensor is equivalent to exposure control. As bothfocus and exposure control subsystems interact with the lens assembly,the imaging sensor and potentially the Bayer-ISP these are shown as asingle system block [114] in FIG. 1.

The image sensor [112] is typically a CMOS image sensor. This comprisesa pixel array ranging in size from 5 Megapixel with the largest in usein consumer products being of the order of 40 Megapixels. Also the sizeof this sensor is important as higher image quality is obtained by usinga larger image sensor. In smartphones a relative small sensor istypically used—this can be as small as 25 sq mm (25 mm²) up to just over100 sq mm (100 mm²); large sensor sizes that are more suitable for theRAW peripheral include 4/3 sensors (225 mm²), APS-C (320-370 mm²) orAPS-H (550 mm²).

In addition to a larger sensor the RAW Module also includes aspecialized image signal processor (ISP) unit [105]. This specializedmodule is optimized to accelerate processing of the RAW image data andin a typical embodiment will provide the various Bayer processingfunctions described by [202] of FIG. 2. In some embodiments thisprocessed Bayer data is transmitted directly to the smartphone but inalternative embodiments it may be further processed to YUV, YCC or RGBcolor spaces. The provided color space and the pre-processing providedwill depend on an initialization process when the smartphone is firstconnected to the module. In some embodiments it may be possible tochange this configuration from the connected smartphone, or by means ofa mode switch on the module, if provided.

The transmitting of an image frame to the connected smartphone ismanaged by a MIPI to USB-3 interface subsystem [116].

The Image Processing Pipeline

This typically refers to a range of image processing that is applied toa RAW (Bayer pattern) image that is obtained from an image sensor. TheBayer pattern is well known to those skilled in the art and employsadditional sensing elements that are responsive to green wavelengths.

The origins of this nomenclature stretch back to Bryce Bayer's patent(U.S. Pat. No. 3,971,065) in 1976 where the inventor refers to the greenphoto-sensors as luminance-sensitive elements and the red and blue onesas chrominance-sensitive elements. He used twice as many green elementsas red or blue to mimic the physiology of the human eye. The luminanceperception of the human retina uses M and L cone cells combined, duringdaylight vision, which are most sensitive to green light. These elementsare referred to as sensor elements or pixel sensors, or simply pixels;sample values sensed by them, after interpolation, become image pixels.

The raw output of Bayer-filter cameras is referred to as a Bayer patternimage. Since each pixel is filtered to record only one of three colors,the data from each pixel cannot fully specify each of the red, green,and blue values on its own. To obtain a full-color image, variousdemosaicing algorithms can be used to interpolate a set of complete red,green, and blue values for each pixel. These algorithms make use of thesurrounding pixels of the corresponding colors to estimate the valuesfor a particular image pixel.

Note that different camera manufacturers have arranged the storage ofBayer data in different, proprietary, file formats. Also, some camerasuse variations on the original Bayer color filter array with differentsensor pixel distributions, or use additional sensor pixels of enhancedsensitivity. The generic term of RAW image data is widely used in thedigital imaging industry to refer to the unprocessed sensor data.Throughout this document when reference is made to Bayer data or RAWimage data the intended scope is any data captured at the full dataresolution (typically 12- or 14-bit) as read out from each of thecamera's image sensor pixels and not the narrower scope as defined inU.S. Pat. No. 3,971,065.

Now, returning to our description of the image processing pipeline,after the demosaicing process a conventional RGB image is provided butthis still requires additional processing. More specifically theadjustment of the image to provide gamma correction, white balance andcolor tone balance. Each of these image adjustments are scene dependentand determining and applying each requires some computation based onstatistical analysis of the acquired image.

The image is then typically converted into YCC or YUV color space asthis facilitates further manipulation of image luminance (Luma) andcolor (Chroma) and eventually image compression to JPEG format. Anexemplary image processing pipeline is illustrated in FIG. 2. In amodern digital imaging device the bulk of the processing pipeline [106]is implemented in a dedicated processing unit known as an image signalprocessor (ISP). The use of a dedicated processing unit allow for theuse of dedicated hardware functions that would normally be provided bysoftware. These dedicated hardware functions can process image data atsignificantly higher rates than a pure software implementation.

In FIG. 2 the ISP pipeline comprises several distinct elements. Firstthe sensor data [104] is input to a first Bayer data processing unit[202] that performs a number of exemplary functions includingcompensating for defect pixels on the sensor, correcting fixed patternnoise, adjusting for lens shading (luminance decreases towards outsideof lens), calculating image statistics (e.g. histograms) and Bayerscaling. This modified Bayer data is next passed to a demosaicing andconversion block [204] that generates first a RGB image, then afteradjustment of the RGB it further converts the image to YCC or YUVformat. A third block [206] performs further adjustments on the YUVsource data and a fourth block is often present and allows imagere-scaling [208] to provide, for example, a preview stream to display onthe screen of the imaging device.

In state-of-art imaging devices all four main processing blocks, [202],[204], [206] and [208] are provided in a single system-on-chip orapplication specific processor known generically as an image signalprocessor (ISP).

In a state-of-art imaging device it is likely that further processing(e.g. high dynamic range (HDR), specialized filters, face and eyetracking, smile detection or red-eye filter) will be performed on themain CPU/GPU of the device. As these algorithms are frequently tuned andadjusted they are more suited to a software implementation rather thanproviding dedicated hardware.

This leads to the underlying inventive concept which is to split theBayer and lower-level image processing into a separate peripheral and bybypassing the internal sensor and ISP of the smartphone and providingRAW image data (or RGB/YUV/YCC in some embodiments) directly to thesmartphone from an external peripheral optimized to provide high-qualityoptical and sensor subsystems similar to those found in DSLR or high-endmirrorless cameras.

Internal Organization of the Smartphone

This is shown in FIG. 5. The smartphone contains a camera module [503]that is typically designed independently of the main phone. It willtypically be interfaced with the main device using MIPI (M-Phy physicalinterface) and in most modules only the lower level Bayer processingcorresponding to [202] in FIG. 2 is provided within the module.

Other processing corresponding to [204], [206] and [208] is typicallyperformed in a dedicated ISP component [516] and a pre-processed RGB orYCC/YUV image is passed on to the main CPU/GPU [509] for more complexanalysis and post-processing. Where such processing is performed it istypically required to store the image, or portions thereof, in atemporary image store [511] and image portions may be moved back andforth via internal data bus [518] between the main CPU/GPU [509] andthis image store. Once processing of the YCC/YUV/RGB image is finalizedit is sent to the compression module [505] and permanent storage [507].

Where additional analysis and processing is not required the image maysimply be sent via internal data bus [514] for compression [505] andpermanent storage [507].

Returning to the smartphone camera module [503] and ISP we note that thestandard for external data connections on mobile devices is typicallythe USB-3 standard. Many smartphone devices incorporate a USB-3 moduleand can be interfaced with external USB-3 peripherals either directly,or using a specialized adapter.

Now US-2013/0191568 “Operating m-phy based communications over universalserial bus (usb) interface, and related cables, connectors, systems andmethods” to Hershko et al. describes how M-Phy (MIPI) interfaces can becontrolled and data transfers achieved via a USB-3 interface.

Thus it will be understood that the ISP [516] in FIG. 5 which normallyaccesses the camera module [503] using MIPI interface to obtain digitalimages could be modified to issue a similar set of MIPI commands to aremote device over the USB-3 bus [501] and to accept digital image datafrom a remote source connected to the same USB-3 bus. A detaileddescription of such technique is given in US-2013/0191568, herebyincorporated by reference.

Thus in a preferred embodiment when the RAW camera peripheral isconnected to the smartphone the ISP is notified and can be configured toaccess the RAW peripheral rather than the internal smartphone cameramodule. This provides high quality RAW images to the smartphone ISP andafter further processing to the main CPU/GPU and compression module foreventual storage [507].

We next explain how manual controls on the RAW peripheral can be used toset and configure the acquisition settings in a number of embodiments.

Manual Controls

In certain embodiments manual controls may be added to the body of themodule to enable manual adjustment of these subsystems, however they mayalso be operated from the smartphone UI as will be explained shortly.Where manual controls are provided an additional microcontroller (notshown) may be added to determine and interpret the state of variousmanual settings and may determine other settings based on look-up tables(LUTs) or calculated from pre-programmed formulae. Where provided thismicrocontroller will also store settings, typically on a dedicatednon-volatile storage and communicate these settings and the configured‘state’ of the module to the module-ISP.

In turn the module ISP will make such information available to theconnected smartphone via the MIPI-to-USB communications link. Thus wherecontrols for manual settings are provided these will update the state ofthe module as communicated to the smartphone, which will, in turn,adjust its UI to reflect the changed state. In some embodiments alow-cost, low power LCD display may be provided and managed by themicrocontroller. This enables the user to set and check acquisitionparameters directly on the module without a need to consult thesmartphone display.

In some embodiments a selection switch is provided that can disable themanual controls on the body of the camera module and allow it to becontrolled directly from the smartphone UI and touchscreen. Inalternative embodiments the decision to enable or disable manualcontrols may be determined by the model of smartphone that is connectedto the peripheral.

The module may also provide an external video-mode, or video-captureswitch.

A mode selection switch similar to that provided on modern cameras thatallows the module to be switched into fully-automatic, semi-automatic(aperture or shutter priority or exposure EV number), and fully manualmodes may also be provided.

In alternative embodiments the camera module may not provide manualcontrols, but can be entirely controlled from the touchscreen interfaceof the smartphone.

In some embodiments the peripheral may include un-numbered adjustmentdials for shutter time, aperture, ISO settings or equivalent. Thesedials allow the values for each setting to be adjusted up(+) or down(−)and the result will be stored in the smartphone app. Thus a photographermay control settings as with a normal camera, using conventional dialseven though the setting values are stored in the correspondingsmartphone app.

Hardware Processing

In a preferred embodiment the camera module contains a basic imagesignal processor that operates only on RAW Bayer images. Some dedicatedhardware enhancements may be provided to support high quality imageacquisition. Typically these will determine one or more characteristicsof an image frame and apply the results to modify the acquisition of afollowing frame.

More specifically focus sharpness may be calculated and provided to anauto-focus subsystem; averaged pixel luminance across blocks or regionsof the image may be determined and used as input to an auto-exposuresubsystem; frame to frame motion can be estimated using integralprojection techniques (summing rows & columns of each image frame) and acombination of these techniques applied to sub-regions of the imageframe can be used to determine frame-to-frame motions, bothtranslational and rotational. When working on raw images the greenchannel is frequently used as equivalent to the luminance channel.

In certain embodiments more advanced hardware techniques may beemployed. For example a basic face detector can be implemented using asmall number of the most inclusive haar classifiers (Cho et al. 2009;Tresadern et al. 2011; Yang et al. 2010) and can provide information onregions of the image; by further separating such classifiers intosymmetric and non-symmetric we can also find partial faces (Corcoran etal. 2013). However the hardware engines being incorporated insmartphones should be employed where practical, thus most of thehardware incorporated into the camera module is targeted to improve rawimage acquisition. Refinement and enhancement of the acquired image willmainly be achieved by the smartphone ISP.

Camera Module Operating Modes

The camera module ISP communicates with the image sensor over a MIPIinterface to acquire a full image frame. This may be initiated by thesmartphone, but may also be initiated by a manual shutter/capture buttonwhere provided. When a full capture has not been initiated the cameramodule defaults to a “preview-stream” mode, unless it has entered asleep state.

Preview Stream Mode

In the preview stream mode the module-ISP continually acquires a lowerresolution image frame—typically matched to the smartphone UI—andtransmits this to the smartphone over the USB-3 interface. If auto-focusis active the focus subsystem adjusts the focus position from frame toframe to achieve the sharpest focus setting. If manual focus is activethen the focus is only changed in response to adjustments of theattached lens. The preview stream from the camera module is furtherprocessed by the smartphone-ISP to provide an RGB/YCC image stream fordisplay on the smartphone touchscreen. Other ‘controls’ may be providedon the touchscreen, allowing the user to manually adjust acquisitionparameters and camera module settings.

The preview stream may be buffered on a temporary basis in the raw imagebuffer on the camera module. Typically this will be only for a limitednumber of image frames and after this fixed number of frames the bufferwill be overwritten. However this allows a history of recent imageframes to be retained and possibly used to enhance the current (latest)image frame without placing a memory load on the smartphone. Selectportions of buffered image frames may be uploaded as required by imageprocessing algorithms on the smartphone.

Metadata from the acquisition subsystems can be saved with thecorresponding image frames in the temporary image store. This includesfocus, exposure and zoom data as well as time & date of acquisition.

In some embodiments it may include metadata associated with hardwareenhancements provided on the camera module or built into the ISP. Forexample, if a hardware face detector was provided this metadata mightindicate if potential face regions were detected in an image and, if so,how many potential face regions were found (overlapping detections maybe counted as just a single face) and the XY locations of potential faceregions.

Other useful hardware functions might include flash-eye detection,calculation of an integral image, integral projection vectors, variousimage histograms, foreground/background and sharpness/contrast maps.

In some embodiments additional hardware subsystems such as GPS may beincluded and thus location data could also be provided.

Initialization

The RAW camera module includes an independent power subsystem, andbattery. It may operate in different embodiments as a USB master, or USBslave but in either case the device is self-powered. In certainembodiments it may include a substantial battery pack, serving as areserve power source for a smartphone.

When the peripheral is first attached to the smartphone the two deviceswill exchange configuration information.

In some preferred embodiments the peripheral will be controlled from adedicated camera app on the smartphone and this app will give access tolow-level system features of both the smartphone, including directaccess to the internal camera module interfaces of the smartphone andthe peripheral via the USB to MIPI interface module.

In alternative embodiments control may be from external cameracontrols—similar to the control knobs provided on a typical DSLRcamera—and the peripheral will communicate these settings to theattached smartphone (which may optionally display them). It is possiblefor the peripheral to provide functionality to operate in both modes,but not at the same time—thus some configuration of the peripheral,either by an external mode switch, or a non-volatile internal state maybe needed.

After the two devices have completed this configuration phase theperipheral will typically be configured in its normal operational modeof image acquisition. The smartphone will also enter a customconfiguration where the low-level image acquisition functions of itscamera subsystem, specifically the optical and sensor components, aredisengaged at the smartphone ISP interface, and the corresponding MIPIinstructions are diverted instead to the USB-3 interface.

The smartphone will typically control normal operational modes of theRAW peripheral through a dedicated camera ‘app’ or application, althoughthese could also be exposed as an API at the OS level. In the lattercase 3^(rd) party ‘apps’ could access the API and thus directly controlthe RAW peripheral.

Image Acquisition Mode

In this mode the various image acquisition settings are programmed onthe peripheral as with any conventional digital camera. The peripheralwill provide a fully automatic mode of image capture, one or moresemi-automatic modes and a full manual mode. Optionally the peripheralmay be programmed to provide more scene-specific ‘smart’ modes, but asthese will typically require image post-processing and are controlledfrom the smartphone they are not considered here.

(i) Full Auto Mode

In the automatic mode the peripheral is configured to implementself-contained auto-focus and exposure of the imaged scene. In this modethe peripheral control sub-systems and ISP continually analyze theacquired preview scene and choose the optimum shutter speed, aperture,ISO and flash settings for image acquisition.

Once set into this mode the peripheral activates and acquires imagesconstantly and adjusts lens, exposure timing and ISP processing tooptimize the acquired scene. Typically this may require some bufferingof image frames to perform frame-to-frame calculations and thus the RAWimage store is accessed by the ISP as required. In some embodiments theacquired image stream will be reduced from full resolution enablinggreater control over the frame-to-frame processing and allowing multipleimage frames to be buffered.

As the user requires to view the imaged scene in order to compose animage the acquired and partly processed (at Bayer level) preview imagestream is transmitted over the USB-3 interface to the attachedsmartphone for display. Additional processing on the smartphone islimited to conversion from Bayer to YCC or similar format suitable forreal-time display on the smartphone screen. Typically the user does nothave direct control over focus, exposure, white balance, or any othercamera parameters, but it may be possible in some embodiments to adjustparameters such as ISO to control the sensitivity of the ‘auto’algorithms.

When the user decides to acquire an image this may be actuated on thesmartphone, or by pressing a capture button on the peripheral. In thefirst case a command is sent via the USB to MIPI interface module toinstruct the ISP to acquire a full resolution RAW image frame andtransmit this to the smartphone via a MIPI to USB transfer. Anyacquisition metadata may also be transferred together with statusinformation immediately preceding or following the image transfer.

In some embodiments where a HDR mode is available a first frame will beacquired and transferred, camera settings may be changed and a secondimage frame is acquired and also transferred together with any relevantimage frame metadata. If some Bayer level processing is employed as partof the process to create a combined HDR image then the first frame maybe temporarily stored in the RAW image store on the peripheral. Imageframe metadata from one or both image frames may optionally betransmitted. In certain embodiments more than two image frames may beacquired to create a HDR image. Typically each frame acquisition will bewith different acquisition parameters, although this is not arequirement and in certain embodiments more than one frame with the samecamera settings may be acquired as references images.

After transmitting to the smartphone the RAW image frames are furtherprocessed by the ISP of the smartphone camera subsystem. This process isessentially the same as if the image had been acquired by thesmartphones camera and sensor, but the smartphone now has access toimproved RAW images with better optical quality due to the use of a DSLRlens unit and a larger full-frame or APS-C image sensor.

(ii) Semi-Automatic Modes

These provides a mode found on DSLR cameras where the user may select aparticular acquisition parameter, e.g. exposure time, aperture, etc. Themost commonly found modes are: AV (Aperture-Priority), TV or S(Shutter-Priority) and P (Programmed Auto).

Aperture-Priority allows the photographer to set the aperture value andthe peripheral automatically sets the correct shutter speed; TV lets thephotographer choose the shutter speed first (for example when shootingsports) and the camera automatically sets the correct aperture.P-Program mode is similar to Auto mode—the shutter and the peripheraldetermines aperture settings, but the photographer can adjust theshooting and image-recording functions.

Alternatively, some of these modes are sometimes presented as beingsuitable for specific scene contexts or shooting conditions. Asexamples:

Portrait Mode (P):

In this mode the peripheral subsystems are configured to assume asubject in the foreground of the frame and choose a shallow depth offield in order to keep the human subject in focus but the backgroundblurred. In low-light situations a flash may be activated.

If the peripheral reads the scene as dark, it may be able to add fill-inflash, increasing the luminance of certain regions of the image. Thismust be implemented at Bayer level and may not be feasible. Inparticular, it is best applied selectively to facial regions andoptionally to strongly shadowed foreground regions. However withimplementation of a basic face-detection or foreground mapping as partof the ISP hardware this can be more effectively implemented.

Landscape Mode:

Typically uses a small aperture (high f/number) to create a well-focusedimage from the foreground into the distance. Landscape mode tends tosuit a wide lens, and again works well if the scene is well lit. Flashis normally disabled.

Sports Mode:

Because sports are fast paced activities, sports mode uses high shutterspeed of at least 1/500- 1/1000 of a second. With a high shutter speedto freeze movement, it means that flash is normally disabled and abrightly lit scene is required. Sports mode can work well alongsidecontinuous shooting mode, where images are taken consecutively—theresult is, for example, a number of shots capturing action in mid air.

Night Portrait:

In the night portrait mode, the peripheral control systems and ISP tryto balance the darkness of the background with the need to light thesubject in the foreground. The aperture will have to be wide to allowsufficient light to capture the background and keep the subject infocus, but at the same time flash is required to illuminate the subjectand avoid blur. Sometimes the night portrait mode will double flash,creating an unusual double exposure look.

These ‘semi-automatic’ modes are distinguished from ‘smart-scene’ modeswhere additional, post-acquisition, image processing is required. Oftenthis employs multiple image frames. Some modern smartphones and camerashave a very long list of such ‘smart’ modes.

In a sense, two-image HDR is an example of such a ‘smart-scene’ butbecause it has become so prevalent in modern smartphones it makes senseto modify the imaging peripheral and Bayer ISP to accommodate thisparticular form of image acquisition.

(iii) Manual Mode

In full manual mode the RAW peripheral does not make any attempts toadjust settings. Instead these are set manually—either by externalswitches and dials on the peripheral itself, or from the smartphone UI.An example of a manual UI for a smartphone is shown in FIG. 6. Thisenables the user to directly set the ISO, white balance, exposure timeand aperture (f-stop) of the camera. Manual adjustment of focus, noisesuppression and optical stabilization may also be provided where suchsubsystems are incorporated into the RAW peripheral.

In response to such settings the peripheral will acquire and process RAWimages and transmit these to the smartphone, thus the user can see theeffects of the various settings from the preview stream transmitted fromperipheral to smartphone. The exact settings may vary for differentembodiments, but will mirror those available on today's DSLR andmirrorless cameras.

There is a particular need for advanced user interfaces that willprovide simplified access to complex camera settings. Some exemplaryembodiments will be presented below.

(iv) ‘Smart’ Scene Modes (with Post-Processing)

These will typically require multi-frame image post-processing and arethus controlled from the smartphone. In such modes the smartphones wouldsimply manipulate the peripheral to capture multiple sequential imageframes, possibly with different acquisition setting as described below.

Some examples include:

Best Face Mode:

This captures multiple image frames of a group of people and allows theuser to select the best face from each; typically it captures at least 5consecutive image frames.

Beauty Modes:

This modifies a portrait image and is mainly based on imagepost-processing, although some features may require two or moreconsecutive image frames to be acquired with different camera settingson the peripheral.

Bokeh and Smart-Portrait Modes:

Bokeh mode blurs the background in a controlled manner and normally itis sufficient to acquire two consecutive image frames with differentfocus settings. Some ‘smart-portrait’ modes can provide Bokeh-like blurwithout a need for the computationally complex blurring required tosimulate lens blur—these typically require 3 consecutive image frames tobe acquired.

Night Portrait or Low-Light Modes:

Basic night portrait mode uses controlled exposure and camera flash tocapture a single image, but it can be improved using separate imageframes with flash and no-flash settings and combining the two.

A range of other flash/no-flash techniques are described in, forexample, “Digital Photography with Flash and No-Flash Image Pairs” byPetschnigg et al from Microsoft Research and available at:http://research.microsoft.com/en-us/um/redmond/projects/flashnoflash/

Another low-light technique involves capturing a blurred image with fullexposure and an unblurred image with short-exposure time. The two imagescan be combined to generate a sharp low-light image.

Focus Stacking or Refocus:

Another technique is to acquire multiple images with different focussetting and combine the ‘in-focus’ parts of each image. In otherexamples these may be stored in a container file providing a means tore-focus an image.

HDR De-Ghosting:

Motion that occurs between the two HDR image frames leads to a ghostlyeffect where a face, or body has moved between the two image frames. Itis possible to remove such ‘ghosts’ by careful and advancedpost-processing but this will be computationally expensive and would notbe implemented on the RAW peripheral described here.

In such ‘smart’ modes the main computation and post-processing will beimplemented on the smartphone, which will typically incorporate amulti-core CPU and GPU units to facilitate such advanced imageprocessing algorithms. The peripheral does not need to provide any ofthe processing for such mode, only to provide the raw images withappropriate settings as requested by the smartphone.

Video Acquisition Mode

Video mode may be selected via the smartphone, or from a switch on theRAW peripheral. Once this mode is activated the peripheral switches tocontinuous acquisition and transfer of image frames to the smartphone.The Bayer ISP is set into a fixed configuration (e.g. ISO, noisesuppression and Bayer processing modes) although it may still beadjusted by the focus and exposure subsystems. And the ISP can continueto provide basic image frame analysis (histogram & focus feedback) tothese subsystems as in the auto-camera mode.

In a preferred embodiment full resolution image frames are transmittedto the smartphone over USB-3 at 30-60 frames per second. Post-processingand compression to an MPEG stream is delegated to the smartphone whichuses the same compression engine as it would for native images. If thevideo resolution supported on the smartphone is less than that providedby the peripheral then ‘full resolution’ from the peripheral may bere-sized to the smartphone resolution before transfer over USB-3.

It is envisaged that at least 1080p HD video would be provided, but 4Kvideo is more likely to become the standard for smartphones ashigh-speed LTE networks are deployed. In any case the video capabilitiesof the peripheral and smartphone would be part of the initializationprocess described previously.

Smartphone Workflows

The peripheral is primarily designed to substitute for the inbuiltcamera of the smartphone at the RAW level. Thus it is optimized for theacquisition and Bayer-level processing of image sensor data and toprovide DSLR equivalent focus, exposure and image frame calibrationcapabilities for DSLR or APS-C sized image sensors.

Image transfers are optimized to minimize transfer lag, and image datacan be transferred directly to the host smartphone with only a small lagas the image bitstream passes through the Bayer ISP on the peripheral.

These raw images are processed on the smartphone using the phone ISP andtreating the Bayer images as if they had originated from the smartphonecamera subsystem. Thus all of the advanced smart-scene processing can beavailable on the smartphone, but using higher quality RAW images due tothe DSLR quality optics and sensor.

Advanced User Interface Embodiments

As mentioned above, most of the higher level imaging capabilities of thesmartphone can still be used, taking advantage of higher image qualityobtained from the RAW module. However, in order to correct set theacquisition parameters for said module when operating in semi-automaticor full manual modes it is necessary to provide the user with simplifiedaccess to these settings.

In this regard FIG. 6 provides an example of a typical user interface inuse by hybrid camera devices such as the Samsung Galaxy camera or theNokia 818 or 1020 camera-phones.

Note the mode selection switches to the right that provide a standardrange of modes used in DSLR or mirrorless cameras. These are summarizedby table 1 below:

TABLE 1 Acquisition modes for a typical DSLR camera. Mode Shutter SpeedAperture P (programmed Selected by camera Selected by camera auto) S(shutter-priority Selected by photographer Selected by camera auto) A(aperture-priority Selected by camera Selected by photographer auto) M(manual) Selected by photographer Selected by photographer

It can be seen that these are quite complex and would be found confusingby a novice user. The different modes are activated by selecting theappropriate switch, for example the ‘S’ mode or shutter-priority can beactuated by switch [609]. The current selected mode is highlighted inthe user interface, in the example of FIG. 6 it is the ‘P’ mode, orprogrammed auto switch [611].

Depending on the selected mode and according to table 1 above the userwill have access to some of the camera acquisition setting dials, namelyISO [603], EV or exposure number [605], f-stop or aperture size [607]and shutter speed [614]. The settings that are auto-selected by thecamera can not be changed and are typically greyed-out or there userinterface elements are shown as inactive. For an inexperienced user thiscan be quite confusing. A further problem is that the user does not havedirect access to the preview of the scene when setting acquisitionparameters from a user interface as shown in FIG. 6.

Accordingly, in FIG. 8 we show an alternative user interfacecorresponding to the vertical alignment of the smartphone. A live-view′preview of the imaged scene is provided on the smartphone screen. Thisis streamed directly from the RAW module and may be optionallysubsampled on the RAW module to reduce resource requirements on thesmartphone device.

It will be seen from FIG. 8 that all of the main camera acquisitionparameters are provided [814] as rotary-dial switched as commonly foundon smartphone user interfaces (e.g. to set alarm times). The user mayconveniently flick through settings and observe the effects on the‘live-view’ image stream. In addition different modes may be selected asshown to the right with S-mode [809], P-mode [802], A-mode [807] andfull manual [811] available. A dedicated video mode is also provided[804]. Typically, in video mode there may be less control over imageacquisition, although this will depend on the underlying capabilitiesand computational resources available both within the RAW module and theconnected smartphone.

In addition a range of programmable ‘f-buttons’ [816] may be provided.These can provide access to a range of advanced filters available on thesmartphone. Examples include image analysis functions such as red-eyefilter, face tracking, beauty filters, smile or blink detection, and soon. These may be programmed by the user, or some may be provided with adedicated function, e.g. flash on/off or red-eye on/off. In someembodiments these switched could cycle through multiple modes of afilter, or select how to combine two or more filters that are commonlyused together.

Finally a specialized main acquisition & control button [820] isprovided to provide access to common adjustments and final acquisitiondecision. This is detailed in FIG. 9 where one embodiment with twoconcentric rings is shown. This has a central active region [901] and atleast one additional concentric ring control [904] with a rotary slider[903]. As it is a software interface element this control can haveadditional rings and in this example a second ring [906] is providedwith corresponding rotary slider [907].

The central active region [901] actuates primary image acquisition, orin video mode it initiates or pauses the record function. Thesurrounding rotary sliders provide means to adjust various cameraparameters, depending on the exact configuration. In a basic examplethese rings can provide intuitive and interactive access to manual lensfocus of the RAW module. Thus moving the rotary slider to the left movesfocus closer to the macro point, to the right focus moves towardsinfinity. The second ring could provide access to a zoom function, wherethe lens module supports optical zoom. Movement of these rotary slidersis converted to MIPI commands and transmitted over the USB interface tothe RAW module where these commands are sent to the lens/focussubsystems. Thus the user can directly access the lens control module ofthe RAW module using these convenient rotary sliders.

A more detailed example of an expanded acquisition & control (A&C)button is shown in FIG. 11. As it is a software component this buttoncan be programmed in a variety of modes to match the various complexsettings of a DSLR camera. According to the relevant mode only a subsetof the acquisition parameters are provided and the number of control‘rings’ will vary accordingly. In the example of FIG. 11 the A&C buttonis modified for shutter priority. Thus it offers, in addition to focusand zoom functionality the ability to control shutter speed [1102] andISO setting [1103]. Each control ring is adjusted by moving the relevantrotary slider [1105].

The advantage of this unified A&C button is shown in FIG. 10. Here wesee that this single control element can be overlaid on a full-screen‘live-view’ image without interfering significantly with the compositionof a photograph. The user may conveniently adjust the various settingprovided by the rotary sliders [1004] and observe the effects in themain ‘live-view’ image [1001].

One additional aspect of the A&C button can be seen from FIG. 10—thecentral acquisition area of the A&C button [1003] can provide ancolor-coded indicator as to the suitability of the acquisition settings.Thus this central area can cycle through a range of colors fromred-orange-yellow-green indicating the likely quality of the finalimage. This also enables the user to learn about the use of differentcamera settings such as shutter speed and aperture and how they effectoverall quality of the acquired image.

By taking advantage of the advanced user-interface capabilities of asmartphone these user interface improvements as described above cangreatly simplify the access of users who are unfamiliar with DSLR andmirrorless cameras to the capabilities of RAW camera module. Claimsrelating to these user interface improvements are appended below.

A handheld imaging peripheral is described to provide high quality RAWimage data. The device incorporates a lens mount that can accommodate aninterchangeable lens unit and a number of external controls to enableimage acquisition settings to be adjusted by the user. The peripheral isconfigured to communicate with a smartphone, or similar handheld device.The peripheral can have a larger dimension, particularly with respect tothe thickness of the handheld device, thus overcoming some of thephysical limitations to providing high quality optics and image sensingwithin the thin form factor of a modern smartphone. By providingexternal controls, similar to those found on high-end digital cameras,the user can experience an improved photographic experience using theirsmartphone to post-process, enhance, display and store the acquiredimages or video. The user avoids the cost and inconvenience ofpurchasing and carrying a separate digital camera and gains theconnectivity and sharing capabilities of the smartphone.

An improved smartphone photographic experience is provided by using anexternal imaging peripheral. The peripheral is connected to thesmartphone via a USB-3, or similar high-speed data bus and is configuredto capture RAW image data of a higher quality than is possible with thesmartphone's camera. The smartphone can be configured to disconnect itsown camera and accept images from the peripheral over the high-speeddata bus. A user interface displayed on the touch-screen of thesmartphone enables the peripheral to be controlled, specifically theacquisition of a next image or activating a video recording. Additionalcontrol functions may include adjustment of the acquisition settings onthe peripheral. Images from the peripheral are analyzed, processed andenhanced on the smartphone. Thus advanced image processing techniquesavailable on the smartphone can be applied to images obtained from theperipheral. Images may be displayed, stored and transmitted over anetwork by the smartphone.

Improved user interfaces are provided to simplify adjustments to theacquisition settings of a camera peripheral from an attached smartphonedevice. In one embodiment a ‘live view’ image stream is provided by theperipheral and displayed on the smartphone display with a range of touchcontrols for adjusting acquisition settings on the peripheral. As theseare adjusted the user may observe the practical effects of theadjustments on the ‘live view’ image. The user may optionally select arange of post-processing filters implemented on the smartphone andobserve the effect of such filters.

In another embodiment a full-screen ‘live view’ is provided on thesmartphone and a single, multi-component, control widget is overlaid onthe ‘live view’. This component incorporates one or more concentricrings with rotary sliders that can be used to adjust a range of settingsfor acquisition parameters dependent on the acquisition mode selectedfor the camera peripheral. User feedback is provided via a centralcircular region that changes color responsive to the ‘goodness’ of theselected acquisition settings. This central region also provides controlmeans to actuate an image acquisition or to commence a video recording.

According to some but not necessarily all embodiments, there isprovided: A handheld imaging device comprising a lens and image sensorfor acquiring digital images, further comprising: a set of externalcontrols provided on the device; an image signal processor configured toperform processing on a Bayer image obtained from said sensor;communication means to transmit said processed Bayer image to a seconddevice; wherein device settings on the external controls are transmittedto the second device, said device further analyzing, processing and/ordisplaying the images transmitted based in part on the settings of theexternal controls.

According to some but not necessarily all embodiments, there isprovided: A peripheral for a handheld imaging device comprising a lensand image sensor for acquiring digital images, further comprising: animage signal processor configured to perform processing on a Bayer imageobtained from said sensor; an interface module configured to convert aMIPI data stream into a USB-3 data stream; and further configured toreceive commands from the handheld imaging device including at least acommand to acquire a next image; and control means within the handheldimaging device to disable its internal imaging subsystem; andcommunication means between handheld imaging device and peripheral toenable the exchange of data and command codes; wherein the handheldimaging device disables its internal imaging subsystem, issues a commandto the peripheral to acquire a next image and transmit said image viathe communication means.

According to some but not necessarily all embodiments, there isprovided: A smartphone peripheral configured to connect to an externalcommunications port on the smartphone providing means to acquire,pre-process and transmit RAW digital image data; further comprising, auser interface means provided on said smartphone, including a‘live-view’ of the imaged scene and control elements to enable selectionand adjustment of a range of acquisition parameters and acquisitionmodes of said peripheral, and communication means to deliver controlcodes to actuate said selections and adjustments, trigger an imageacquisition and transmit the acquired image to the smartphone, wherein,prior to image acquisition said peripheral continuously acquires images,said communications means continuously transmits images and saidsmartphone continuously receives and displays said images to provide theuser with a ‘live view’ display.

According to some but not necessarily all embodiments, there isprovided: A touch sensitive user interface element comprising: a centralactuation region of approximately circular shape; one or more ringsconcentric with said actuation region, each equipped with at least onerotary slider; wherein, at least one of the rotary sliders performs afocus or zoom adjustment in a separate peripheral configured to acquiredigital images, and the interface element is displayed on a smartphonetouchscreen, said phone being communicatively coupled to saidperipheral.

According to some but not necessarily all embodiments, there isprovided: A handheld imaging device comprising a lens and image sensorfor acquiring digital images, further comprising: an image signalprocessor configured to perform processing on a Bayer image obtainedfrom said sensor; an interface module configured to convert a MIPI datastream into a USB-3 data stream; and further configured to receivecommands from a second USB-3 device and to transmit image metadata,including at least image acquisition parameters associated with a Bayerimage to said second device.

According to some but not necessarily all embodiments, there isprovided: A handheld image capture peripheral, comprising: a lens and/ora lens mount configured to attach a lens; an optical sensor configuredto capture RAW image data; and an I/O interface configured to transmitsaid RAW image data out of the peripheral, said RAW image data not beingdemosaiced prior to said transmission.

According to some but not necessarily all embodiments, there isprovided: A handheld image capture peripheral, comprising: a lens and/ora lens mount configured to attach a lens; an optical sensor configuredto capture RAW image data, using said lens or an attached lens, in atleast partial dependence on one or more image acquisition parameterscomprising one or more of focus, exposure and zoom settings; and an I/Ointerface configured to transmit out of the peripheral said RAW imagedata with image metadata that describes said RAW image data and includesat least said image acquisition parameters.

According to some but not necessarily all embodiments, there isprovided: A handheld image capture peripheral, comprising: a lens and/ora lens mount configured to attach a lens; an optical sensor configuredto capture RAW image data; an image signal processor configured topre-process said RAW image data; and an I/O interface configured totransmit out of said peripheral said pre-processed RAW image data,wherein said pre-processing comprises one or more of: optical blackcompensation, pixel defect compensation, fixed pattern noisecompensation, lens shading compensation, gains and offsets adjustment,3A statistics generation and storage, Bayer scaling, and image resizing.

According to some but not necessarily all embodiments, there isprovided: A handheld mobile processing unit, comprising: a cameracomprising a lens and a sensor; an I/O interface configured to transmitand receive data and commands into and out of the mobile unit; and animage signal processor configured to receive RAW image data from saidcamera and process said image data into a display-ready format; whereinsaid image signal processor is configured, on receipt of a command touse a physically distinct imager to capture an image, to use said I/Ointerface to transmit said capture-image command out of the mobile unit,and to process RAW image data received via said I/O interface inresponse to said command transmission into a display-ready format.

According to some but not necessarily all embodiments, there isprovided: A handheld mobile processing unit, comprising: a cameracomprising a lens and a sensor; a display; an I/O interface configuredto transmit and receive data and commands; and an image signal processorconfigured to receive and process image data captured by said camera, toprocess into display-ready format a RAW video image data stream receivedby said I/O interface from a device physically distinct from thehandheld mobile unit, and to cause said display to display saidprocessed video stream live with respect to said receipt of said videoimage data stream; wherein said RAW video image data stream is notdemosaiced prior to said receipt.

According to some but not necessarily all embodiments, there isprovided: A handheld mobile processing unit, comprising: a cameracomprising a lens and a sensor; an I/O interface configured to transmitcommands to and receive data from a device physically distinct from themobile unit; and an image signal processor configured to receive RAWimage data from said camera and process said image data into adisplay-ready format; wherein said image signal processor is configured,on receipt of a command to use a physically distinct imager to capturean image, to use said I/O interface to transmit said capture-imagecommand to the physically distinct device, and to process RAW image datareceived via said I/O interface in response to said command transmissioninto a display-ready format in at least partial dependence on said imageacquisition parameters; and at least one of: wherein said I/O interfaceis configured to enable user selection of image acquisition parameters,wherein said image signal processor is configured to use said I/Ointerface to transmit said image acquisition parameters to saidphysically distinct device on receipt of said capture-image command; andwherein said image signal processor is configured to perform saidprocessing using image acquisition parameters received from said I/Ointerface with said RAW image data and not selected using the mobileunit.

According to some but not necessarily all embodiments, there isprovided: An image capture and processing system, comprising: a handheldperipheral image capture unit, comprising: a peripheral lens and/or alens mount configured to attach a peripheral lens; a peripheral sensorconfigured to capture RAW image data, said peripheral sensor having alarger sensing surface than said mobile unit sensor; a peripheral I/Ointerface configured to transmit at least said RAW image data, said RAWimage data not having been demosaiced, to said handheld mobile I/Ointerface; and a handheld mobile processing unit physically distinctfrom said handheld peripheral image capture unit, comprising: a cameracomprising a mobile lens and a mobile sensor; a mobile I/O interfaceconfigured to receive at least said RAW image data from said handheldperipheral I/O interface; and a mobile image signal processor configuredto process said RAW image data into a display-ready form.

According to some but not necessarily all embodiments, there isprovided: An image capture and processing system, comprising: a handheldperipheral image capture unit, comprising: a peripheral lens and/or alens mount configured to attach a peripheral lens; a peripheral sensorconfigured to capture a RAW video image data stream; a peripheral I/Ointerface configured to transmit live at least said RAW video image datastream, said RAW video image data stream not having been demosaiced, tosaid handheld mobile I/O interface; and a handheld mobile processingunit physically distinct from said handheld peripheral image captureunit, comprising: a camera comprising a mobile lens and a mobile sensor;a display; a mobile I/O interface configured to receive at least saidRAW video image data stream from said handheld peripheral I/O interface;and a mobile image signal processor configured to process said RAW videoimage data stream into a display-ready form and to cause said display todisplay said display-ready video stream live with respect to saidcapture action.

According to some but not necessarily all embodiments, there isprovided: An image capture and processing system, comprising: a handheldperipheral image capture unit, comprising: a peripheral lens and/or alens mount configured to attach a peripheral lens; a peripheral sensorconfigured to capture RAW image data in at least partial dependence onimage acquisition parameters comprising one or more of focus, exposureand zoom; a peripheral I/O interface configured to transmit at leastsaid RAW image data, said RAW image data not having been demosaiced, tosaid handheld mobile I/O interface; and a handheld mobile processingunit physically distinct from said handheld peripheral image captureunit, comprising: a camera comprising a mobile lens and a mobile sensor;a mobile I/O interface configured to receive at least said RAW imagedata from said handheld peripheral I/O interface; and a mobile imagesignal processor configured to process said RAW image data into adisplay-ready form in at least partial dependence on said imageacquisition parameters; wherein at least one of said mobile I/Ointerface and said peripheral I/O interface is configured to enable userselection of image acquisition parameters and to transmit saidparameters to the other of said mobile and said peripheral I/Ointerfaces.

According to some but not necessarily all embodiments, there isprovided: A method of capturing and processing images, comprising:capturing RAW image data using a sensor and a lens of a handheldperipheral; transmitting said RAW image data from said peripheral to ahandheld mobile processing unit physically distinct from said handheldperipheral without demosaicing said RAW image data; and processing saidRAW image data into a display-ready format using an image signalprocessor of said handheld mobile processing unit.

According to some but not necessarily all embodiments, there isprovided: A method of capturing and processing images, comprising:capturing a RAW image data stream using a sensor and a lens of ahandheld peripheral; transmitting said RAW image data stream from saidhandheld peripheral to a handheld mobile processing unit physicallydistinct from said handheld peripheral without demosaicing said RAWimage data stream; processing said RAW image data into a display-readyformat using an image signal processor of said handheld mobileprocessing unit; and displaying said display-ready image data stream asvideo using a display of said handheld mobile processing unit, whereinsaid transmitting and said processing are performed such that said videois displayed live with respect to said capturing.

According to some but not necessarily all embodiments, there isprovided: A method of capturing and processing images, comprising:capturing RAW image data using a sensor and a lens of a handheldperipheral; transmitting said RAW image data from said peripheral to ahandheld mobile processing unit physically distinct from said handheldperipheral without demosaicing said RAW image data; processing said RAWimage data into a display-ready format using an image signal processorof said handheld mobile processing unit; and at least one of: selectingimage acquisition parameters using said handheld mobile processing unitand parameter-transmitting said parameters to said handheld peripheral,wherein said capturing is performed in at least partial dependence onsaid parameters; and selecting image acquisition parameters using saidhandheld peripheral, wherein said transmitting transmits said parameterswith said RAW image data; wherein said parameters comprise one or moreof focus, exposure and zoom.

Those of skill in the art would further appreciate that the variousillustrative logical blocks, modules, circuits, and algorithms describedin connection with the embodiments disclosed herein may be implementedas electronic hardware, instructions stored in memory or in anothercomputer-readable medium and executed by a processor or other processingdevice, or combinations of both. The arbiters, master devices, and slavedevices described herein may be employed in any circuit, hardwarecomponent, integrated circuit (IC), or IC chip, as examples. Memorydisclosed herein may be any type and size of memory and may beconfigured to store any type of information desired. To clearlyillustrate this interchangeability, various illustrative components,blocks, modules, circuits, and steps have been described above generallyin terms of their functionality. How such functionality is implementeddepends upon the particular application, design choices, and/or designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed with a processor, a DSP, an Application Specific IntegratedCircuit (ASIC), an FPGA or other programmable logic device, discretegate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A processor may be a microprocessor, but in the alternative, theprocessor may be any conventional processor, controller,microcontroller, or state machine. A processor may also be implementedas a combination of computing devices, e.g., a combination of a DSP anda microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

The embodiments disclosed herein may be embodied in hardware and ininstructions that are stored in hardware, and may reside, for example,in Random Access Memory (RAM), flash memory, Read Only Memory (ROM),Electrically Programmable ROM (EPROM), Electrically ErasableProgrammable ROM (EEPROM), registers, hard disk, a removable disk, aCD-ROM, or any other form of computer readable medium known in the art.An exemplary storage medium is coupled to the processor such that theprocessor can read information from, and write information to, thestorage medium. In the alternative, the storage medium may be integralto the processor. The processor and the storage medium may reside in anASIC. The ASIC may reside in a remote station. In the alternative, theprocessor and the storage medium may reside as discrete components in aremote station, base station, or server.

It is also noted that the operational steps described in any of theexemplary embodiments herein are described to provide examples anddiscussion. The operations described may be performed in numerousdifferent sequences other than the illustrated sequences. Furthermore,operations described in a single operational step may actually beperformed in a number of different steps. Additionally, one or moreoperational steps discussed in the exemplary embodiments may becombined. It is to be understood that the operational steps illustratedin the flow chart diagrams may be subject to numerous differentmodifications as will be readily apparent to one of skill in the art.Those of skill in the art would also understand that information andsignals may be represented using any of a variety of differenttechnologies and techniques. For example, data, instructions, commands,information, signals, bits, symbols, and chips that may be referencedthroughout the above description may be represented by voltages,currents, electromagnetic waves magnetic fields or particles, opticalfields or particles, or any combination thereof.

The previous description of the disclosure is provided to enable anyperson skilled in the art to make or use the disclosure. Variousmodifications to the disclosure will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other variations without departing from the spirit or scopeof the disclosure. Thus, the disclosure is not intended to be limited tothe examples and designs described herein, but is to be accorded thewidest scope consistent with the principles and novel features disclosedherein.

REFERENCES

-   [1] C. Dainty, “Film Photography Is Dead: Long Live Film: What can    Digital Photography Learn from the Film Era?,” IEEE Consum.    Electron. Mag., vol. 1, no. 1, pp. 61-64, January 2012 [Online].    Available:    http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6107480.    [Accessed: 29 Jul. 2013]-   [2] J. Cho, S. Mirzaei, J. Oberg, and R. Kastner, “Fpga-based face    detection system using Haar classifiers,” in Proceeding of the    ACM/SIGDA international symposium on Field programmable gate    arrays—FPGA '09, 2009, p. 103 [Online]. Available:    http://dLacm.org/citation.cfm?id=1508128.1508144. [Accessed: 28 Oct.    2011]-   [3] P. A. Tresadern, M. C. Ionita, and T. F. Cootes, “Real-Time    Facial Feature Tracking on a Mobile Device,” Int. J. Comput. Vis.,    pp. 1-10, May 2011 [Online]. Available:    http://www.springerlink.com/content/a0h707650870942h/. [Accessed: 1    Dec. 2011]-   [4] M. Yang, J. Crenshaw, B. Augustine, R. Mareachen, and Y. Wu,    “AdaBoost-based face detection for embedded systems,” Comput. Vis.    Image Underst., vol. 114, no. 11, pp. 1116-1125, November 2010    [Online]. Available: http://dx.doi.org/10.1016/j.cviu.2010.03.010.    [Accessed: 21 Jul. 2011]-   [5] P. M. Corcoran, P. Bigioi, and F. Nanu, “Half-face detector for    enhanced performance of flash-eye filter,” in 2013 IEEE    International Conference on Consumer Electronics (ICCE), 2013, pp.    252-253 [Online]. Available:    http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6486882.    [Accessed: 7 Oct. 2013]

1-40. (canceled)
 41. A handheld mobile processing unit, comprising: acamera comprising a lens and a sensor; an Input/Output (I/O) interfaceconfigured to transmit and receive data and commands into and out of themobile unit; and an image signal processor configured to receive RAWimage data from said camera and process said image data into adisplay-ready format; wherein said image signal processor is configured,on receipt of a command to use a physically distinct imager to capturean image, to use said I/O interface to transmit said capture-imagecommand out of the mobile unit, and to process un-demosaicked RAW imagedata received via said I/O interface in response to said commandtransmission into a display-ready format.
 42. The handheld mobileprocessing unit of claim 41, further comprising a telephone, wherein themobile unit is a smartphone.
 43. The handheld mobile processing unit ofclaim 41, further comprising a display, said image signal processorconfigured to cause said display to display said display-ready imagedata.
 44. The handheld mobile processing unit of claim 41, furthercomprising a controller configured to disable said camera on receipt ofsaid capture-image command.
 45. The handheld mobile processing unit ofclaim 41, wherein said image signal processor is configured to access aninternal bus to receive said RAW image data directly from said I/Ointerface.
 46. The handheld mobile processing unit of claim 41, whereinsaid display-ready image data comprises compressed or uncompressednormalized RGB image data.
 47. The handheld mobile processing unit ofclaim 41, wherein said image signal processor is configured to causesaid I/O interface to transmit commands specifying image acquisitionparameters with said capture-image command, said image acquisitionparameters including one or more of a physical aperture setting and aphysical focus setting.
 48. A handheld mobile processing unit,comprising: a camera comprising a lens and a sensor; a display; anInput/Output (I/O) interface configured to transmit and receive data andcommands; and an image signal processor configured to receive andprocess image data captured by said camera, to process intodisplay-ready format a RAW video image data stream received by said I/Ointerface from a device physically distinct from the handheld mobileunit, and to cause said display to display said processed video streamlive with respect to said receipt of said video image data stream;wherein said RAW video image data stream is not demosaiced prior to saidreceipt.
 49. The handheld mobile processing unit of claim 48, whereinsaid I/O interface is configured to transmit a user-entered command tosaid physically distinct device to capture one or more images, and saidimage signal processor is configured to process RAW image data receivedby said I/O interface in response to said capture-image commandtransmission.
 50. The handheld mobile processing unit of claim 48,wherein said image signal processor is configured to deactivate saidcamera when said capture-image command is entered.
 51. The handheldmobile processing unit of claim 48, wherein said image signal processoris configured to deactivate said camera during said receipt and/orprocessing of said video image data stream.
 52. The handheld mobileprocessing unit of claim 48, wherein said image signal processor isconfigured to access an internal bus to receive said RAW image datadirectly from said I/O interface.
 53. The handheld mobile processingunit of claim 48, wherein said display-ready image data comprisescompressed or uncompressed normalized RGB image data.
 54. The handheldmobile processing unit of claim 48, wherein said image signal processoris configured to cause said I/O unit to transmit commands specifyingimage acquisition parameters with said capture-image command, said imageacquisition parameters including one or more of a physical aperturesetting and a physical focus setting.
 55. A handheld mobile processingunit, comprising: a camera comprising a lens and a sensor; anInput/Output (I/O) interface configured to transmit commands to andreceive data from a device physically distinct from the mobile unit; andan image signal processor configured to receive RAW image data from saidcamera and process said image data into a display-ready format; whereinsaid image signal processor is configured, on receipt of a command touse a physically distinct imager to capture an image, to use said I/Ointerface to transmit said capture-image command to the physicallydistinct device, and to process un-demosaicked RAW image data receivedvia said I/O interface in response to said command transmission into adisplay-ready format in at least partial dependence on said imageacquisition parameters; and at least one of: wherein said I/O interfaceis configured to enable user selection of image acquisition parameters,wherein said image signal processor is configured to use said I/Ointerface to transmit said image acquisition parameters to saidphysically distinct device on receipt of said capture-image command; andwherein said image signal processor is configured to perform saidprocessing using image acquisition parameters received from said I/Ointerface with said un-demosaicked RAW image data and not selected usingthe mobile unit.
 56. The handheld mobile processing unit of claim 55,further comprising a telephone, wherein the mobile unit is a smartphone.57. The handheld mobile processing unit of claim 55, further comprisinga display, said image signal processor configured to cause said displayto display said display-ready image data.
 58. The handheld mobileprocessing unit of claim 55, further comprising a controller configuredto disable said camera on receipt of said capture-image command.
 59. Thehandheld mobile processing unit of claim 55, wherein said image signalprocessor is configured to access an internal bus to receive said RAWimage data directly from said I/O interface.
 60. The handheld mobileprocessing unit of claim 55, wherein said display-ready image datacomprises compressed or uncompressed normalized RGB image data. 61-100.(canceled)